Image processing method and system

ABSTRACT

The present application relates to an image processing method and system. The method includes: determining an enhanced image of a target object of an input image based on a segmentation algorithm, where the enhanced image of the target object comprises an image in which each pixel classified as the target object is displayed in an enhanced manner; and determining a positioning image of the target object by applying an integral image algorithm to the enhanced image of the target object.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of International Patent ApplicationNo. PCT/CN2021/136052, filed on Dec. 7, 2021, the disclosure of which isincorporated by reference in its entirety.

TECHNICAL FIELD

The present application relates to computer technologies, and inparticular, to image processing technologies.

BACKGROUND

Image processing with computers is widely used in various fields. Imageprocessing can be used to improve the visual quality of images, extractfeatures of specific objects in images, store and transmit images, andthe like. To extract a feature of a specific object in an image, it isdesirable to identify and position the specific object.

Therefore, an improved technology that can accurately position aspecific object in an image is required.

SUMMARY

In view of the above problems, the present application provides an imageprocessing method and system that can improve the accuracy ofpositioning and segmenting a specific target in an image.

According to a first aspect, the present application provides an imageprocessing method, including: determining an enhanced image of a targetobject of an input image based on a segmentation algorithm, where theenhanced image of the target object includes an image in which eachpixel classified as the target object is displayed in an enhancedmanner; and determining a positioning image of the target object byapplying an integral image algorithm to the enhanced image of the targetobject.

In the technical solution of this embodiment of the present application,whether each pixel in the image belongs to the target object or belongsto the non-target object is classified based on the segmentationalgorithm, and on this basis, the target object is positioned in theinput image. The segmentation and positioning of the target object arecombined and the segmentation algorithm and the integral image algorithmare combined, so that the positioning accuracy of the target object canbe improved.

In some embodiments, the determining an enhanced image of a targetobject of an input image based on a segmentation algorithm furtherincludes: performing feature extraction on the input image to determinea pixel feature graph; performing feature extraction on the input imageto determine a context feature graph; determining context relationinformation of each pixel based on the pixel feature graph and thecontext feature graph; and determining the enhanced image of the targetobject according to the context relation information and the inputimage, where a pixel of the enhanced image of the target object includesweight information, and the weight information is related to whether thepixel belongs to the target object. In the present application, not onlypixel-level classification information, but also context classificationinformation around a target pixel are considered in the segmentationalgorithm, and a final classification result of the target pixel isdetermined based on the relation between the target pixel and thecontext thereof. Context information is incorporated into a segmentationalgorithm to further improve the classification accuracy of the targetpixel, thereby segmenting the target object more accurately. A weightapplied to each pixel finally classified as the target object is changedto generate the enhanced image of the target object, so that the targetobject is displayed in an enhanced manner, thereby providing a moreaccurate basis for subsequent positioning processing and furtherimproving the positioning accuracy of the target object. The weight maybe configured by a user. A change of weight setting can affect theenhancement effect of the target object in the enhanced image of thetarget object, so that the desired enhancement effect of the targetobject can be achieved through user setting.

In some embodiments, the determining a positioning image of the targetobject by applying an integral image algorithm to the enhanced image ofthe target object further includes: determining an integral imageaccording to the enhanced image of the target object; and determiningthe positioning image of the target object by using the integral image.The integral image algorithm is applied to the enhanced image of thetarget object in which the target object is displayed in an enhancedmanner, so that the positioning accuracy of the target object can befurther improved.

In some embodiments, the determining an integral image according to theenhanced image of the target object further includes: applying a scalefactor to the enhanced image of the target object. A volume of data tobe processed can be adjusted by applying a scale factor, so that anoperation process can be accelerated and/or the accuracy of the integralimage can be improved according to actual needs.

In some embodiments, the method further includes: calculating a lossrate between the enhanced image of the target object and the input imagebased on a loss function; and feeding back the calculated loss rate tothe segmentation algorithm. A loss rate between the enhanced image ofthe target object output by the segmentation algorithm and a labeledproduction line image reflects the similarity between the enhanced imageof the target object output by the segmentation algorithm and theoriginal input image. The loss rate is fed back to the segmentationalgorithm to perform supervised learning training on the segmentationalgorithm. The accuracy of the segmentation algorithm can be improvedthrough continuous training and learning while fit regression istrained.

In some embodiments, the method further includes: updating thesegmentation algorithm based on the loss rate or the labeled productionline image or a combination of both. The segmentation algorithm in thepresent application trains the calculated loss rate or the labeledproduction line image or a combination of both as training data, and cancontinuously improve the accuracy of the segmentation algorithm intarget object segmentation in a supervised learning manner. In addition,since the training data all comes from the real production line, thetraining data can cover actual needs and be used and promoted inpractice in the production line.

In some embodiments, the segmentation algorithm is implemented by a deepconvolutional neural network HRNet18. HRNet18 maintains high-resolutionfeatures in the entire segmentation algorithm process and facilitatesaccurate segmentation of the target object. In addition, differentbranches of the HRNet18 network produce features of differentresolutions, and these features interact to obtain information, so thathigh-resolution features including multi-channel information can beobtained. In addition, in a case of a limited volume of training data,selection of the HRNet18 model avoids the risk of overfitting and at thesame time can accelerate the operation speed of the entire segmentationalgorithm because of the small structure of the model.

According to a second aspect, the present application provides an imageprocessing system, including: a segmentation module, configured todetermine an enhanced image of a target object of an input image basedon a segmentation algorithm, where the enhanced image of the targetobject includes an image in which each pixel classified as the targetobject is displayed in an enhanced manner; and a positioning imagegeneration module, configured to determine a positioning image of thetarget object by applying an integral image algorithm to the enhancedimage of the target object.

In the technical solution of this embodiment of the present application,whether each pixel in the image belongs to the target object or belongsto the non-target object is classified based on the segmentationalgorithm, and on this basis, the target object is positioned in theinput image. The segmentation and positioning of the target object arecombined and the segmentation algorithm and the integral image algorithmare combined, so that the positioning accuracy of the target object canbe improved.

In some embodiments, the segmentation module further includes: a featureextraction component, configured to perform feature extraction on theinput image to determine a pixel feature graph and perform featureextraction on the input image to determine a context feature graph; acontext component, configured to determine context relation informationof each pixel based on the pixel feature graph and the context featuregraph; and an enhanced image generation component, configured todetermine the enhanced image of the target object according to thecontext relation information and the input image, where a pixel of theenhanced image of the target object includes weight information, and theweight information is related to whether the pixel belongs to the targetobject. In the present application, not only pixel-level classificationinformation, but also context classification information around a targetpixel are considered in the segmentation algorithm, and a finalclassification result of the target pixel is determined based on therelation between the target pixel and the context thereof. Contextinformation is incorporated into a segmentation algorithm to furtherimprove the classification accuracy of the target pixel, therebysegmenting the target object more accurately. A weight applied to eachpixel finally classified as the target object is changed to generate theenhanced image of the target object, so that the target object isdisplayed in an enhanced manner, thereby providing a more accurate basisfor subsequent positioning processing and further improving thepositioning accuracy of the target object. The weight may be configuredby a user. A change of weight setting can affect the enhancement effectof the target object in the enhanced image of the target object, so thatthe desired enhancement effect of the target object can be achievedthrough user setting.

In some embodiments, the positioning image generation module is furtherconfigured to: determine an integral image according to the enhancedimage of the target object; and determine the positioning image of thetarget object by using the integral image. The integral image algorithmis applied to the enhanced image of the target object in which thetarget object is displayed in an enhanced manner, so that thepositioning accuracy of the target object can be further improved.

In some embodiments, the positioning image generation module is furtherconfigured to apply a scale factor to the enhanced image of the targetobject. A volume of data to be processed can be adjusted by applying ascale factor, so that an operation process can be accelerated and/or theaccuracy of the integral image can be improved according to actualneeds.

In some embodiments, the system further includes a loss rate module,configured to: calculate a loss rate between the enhanced image of thetarget object and the input image based on a loss function; and feedback the calculated loss rate to the segmentation module. A loss ratebetween the enhanced image of the target object output by thesegmentation algorithm and a labeled production line image reflects thesimilarity between the enhanced image of the target object output by thesegmentation algorithm and the original input image. The loss rate isfed back to the segmentation algorithm to perform supervised learningtraining on the segmentation algorithm. The accuracy of the segmentationalgorithm can be improved through continuous training and learning whilefit regression is trained.

In some embodiments, the segmentation module is further configured toupdate the segmentation module based on the loss rate or the labeledproduction line image or a combination of both. The segmentationalgorithm in the present application trains the calculated loss rate orthe labeled production line image or a combination of both as trainingdata, and can continuously improve the accuracy of the segmentationalgorithm in target object segmentation in a supervised learning manner.In addition, since the training data all comes from the real productionline, the training data can cover actual needs and be used and promotedin practice in the production line.

According to a third aspect, the present application provides an imageprocessing system, including: a memory storing computer-executableinstructions; and a processor coupled to the memory, where thecomputer-executable instructions, when executed by the processor, causethe system to perform the following operations: determining an enhancedimage of a target object of an input image based on a segmentationalgorithm, where the enhanced image of the target object includes animage in which each pixel classified as the target object is displayedin an enhanced manner; and determining a positioning image of the targetobject by applying an integral image algorithm to the enhanced image ofthe target object.

In the technical solution of this embodiment of the present application,whether each pixel in the image belongs to the target object or belongsto the non-target object is classified based on the segmentationalgorithm, and on this basis, the target object is positioned in theinput image. The segmentation and positioning of the target object arecombined and the segmentation algorithm and the integral image algorithmare combined, so that the positioning accuracy of the target object canbe improved.

The aforementioned description is only an overview of the technicalsolutions of the present application. In order to more clearlyunderstand the technical means of the present application to implementsame according to the contents of the specification, and in order tomake the aforementioned and other objects, features and advantages ofthe present application more obvious and understandable, specificembodiments of the present application are exemplarily described below.

BRIEF DESCRIPTION OF THE DRAWINGS

Various other advantages and benefits will become apparent to those ofordinary skill in the art upon reading the following detaileddescription of preferred embodiments. The drawings are merely for thepurpose of illustrating the preferred embodiments and are not to beconstrued as limiting the present application. Moreover, like componentsare denoted by like reference numerals throughout the drawings. In thedrawings:

FIG. 1 is a flowchart of an image processing method according to someembodiments of the present application;

FIG. 2 is a flowchart of a method for determining an enhanced image of atarget object of an input image based on a segmentation algorithmaccording to some embodiments of the present application;

FIG. 3 is an effect diagram of steps of segmenting a target objectaccording to some embodiments of the present application;

FIG. 4 is an effect diagram of steps of positioning a target objectaccording to some embodiments of the present application;

FIG. 5 is an architectural diagram of a network model for implementing asegmentation algorithm of an image processing method according to someembodiments of the present application;

FIG. 6 is a functional block diagram of an image processing systemaccording to some embodiments of the present application;

FIG. 7 is a functional block diagram of a segmentation module accordingto some embodiments of the present application; and

FIG. 8 is a structural block diagram of a computer system suitable forimplementing an image processing system according to some embodiments ofthe present application.

DETAILED DESCRIPTION

Embodiments of the technical solutions of the present application willbe described in more detail below with reference to the drawings. Thefollowing embodiments are merely intended to more clearly illustrate thetechnical solutions of the present application, so they merely serve asexamples, but are not intended to limit the scope of protection of thepresent application.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meanings as those commonly understood by those skilled inthe art to which the present application belongs. The terms used hereinare merely for the purpose of describing specific embodiments, but arenot intended to limit the present application. The terms “comprising”and “having” and any variations thereof in the description and theclaims of the present application as well as the brief description ofthe accompanying drawings described above are intended to covernon-exclusive inclusion.

In the description of the embodiments of the present application, thetechnical terms “first”, “second”, etc. are merely used fordistinguishing different objects, and are not to be construed asindicating or implying relative importance or implicitly indicating thenumber, particular order or primary-secondary relationship of thetechnical features modified thereby. In the description of theembodiments of the present application, the phrase “multiple” means twoor more, unless otherwise explicitly and specifically defined.

The phrase “embodiment” mentioned herein means that the specificfeatures, structures, or characteristics described in conjunction withthe embodiment can be encompassed in at least one embodiment of thepresent application. The phrase at various locations in the descriptiondoes not necessarily refer to the same embodiment, or an independent oralternative embodiment exclusive of another embodiment. Those skilled inthe art understand explicitly or implicitly that the embodimentdescribed herein may be combined with another embodiment.

In the description of the embodiments of the present application, theterm “and/or” is merely intended to describe the associated relationshipof associated objects, indicating that three relationships can exist,for example, A and/or B can include: the three instances of A alone, Aand B simultaneously, and B alone. In addition, the character “/” hereingenerally indicates an “or” relationship between the associated objects.

In the description of the embodiments of the present application, theterm “multiple” means two or more (including two), similarly the term“multiple groups” means two or more groups (including two groups), andthe term “multiple pieces” means two or more pieces (including twopieces).

Image processing with computers is widely used in various fields. Imageprocessing can be used to improve the visual quality of images, extractfeatures of specific objects in images, store and transmit images, andthe like. To extract a feature of a specific object in an image, it isdesirable to identify and position the specific object. The extractionof the specific object can be used for defect detection of the specificobject. For example, for power lithium batteries, images of lithiumbatteries produced on a production line are captured and target objectssuch as tabs are positioned, so that it can be effectively detectedwhether the tabs have defects such as folding.

In a production process of power lithium batteries, defects areinevitable due to process and device reasons. In each procedure of theproduction line, it is crucial to detect whether tabs of lithiumbatteries are folded. The validity of a detection result ensures thesafety of batteries when batteries are delivered from factories.However, since a tab only occupies a very small percentage of an entirelithium battery, the detection of whether the tab is folded has veryhigh requirements on a resolution of an image and accurate positioningof the tab.

Some image processing methods include: performing double-Gaussiandifference on an input image, labeling the processed image, constructinga neural network and a model for training and learning, and finallyperforming data inference based on the model. In such technologies, thefirst step is generally inputting image data into the model for featureextraction. Therefore, the quality (for example, a resolution and asignal-to-noise ratio) of the input image data directly affects theaccuracy of the trained model. When a target object such as a lithiumbattery tab is small, the method of double-Gaussian difference cannoteffectively position an extremely small target object that requires anextremely high resolution, and image background (a non-target object)has a large interference on the target object. This leads to lowaccuracy of target object positioning and ultimately leads to thedifficulty in accurately detecting defects of the target object (forexample, whether the tab is folded). Therefore, an improved technologythat can accurately position a target object that occupies a smallpercentage of an image and requires a high resolution is required.

In view of the above problems, the present application provides atechnology that can accurately position a target object that occupies asmall percentage of an image and requires a high resolution. Thesolution of the present application may include segmentation of thetarget object and positioning of the target object. In a segmentationstage, in the present application, an enhanced image of a target objectof an input image is determined based on a segmentation algorithm, wherethe enhanced image of the target object includes an image in which eachpixel classified as the target object is displayed in an enhancedmanner; In a positioning stage, in the present application, an integralimage is generated according to the enhanced image of the target object,and a positioning image of the target object is generated based on anintegral image algorithm.

In the technical solution of this embodiment of the present application,whether each pixel in the image belongs to the target object or belongsto the non-target object is classified based on the segmentationalgorithm, and on this basis, the target object is positioned in theinput image. The segmentation and positioning of the target object arecombined and the segmentation algorithm and the integral image algorithmare combined, so that the positioning accuracy of the target object canbe improved.

The technical solutions of the embodiments of the present applicationare applicable to segmentation and positioning of a target object thatoccupies a small percentage of an image and requires a high resolution,including but not limited to: defect detection of tabs in lithiumbatteries, recognition and labeling of species observed in the wild,detection and interpretation of human facial micro-expressions, and thelike. In a case of observing a species in the wild, recognition of thespecies is generally based on labeling of specific patterns and stripeson a specific part of its face or body, and an infrared camera forobservation in the wild often cannot provide a clear high-resolutionimage. Therefore, improved segmentation and positioning algorithms inthe present application are used to improve segmentation and positioningof specific patterns and stripes, which facilitates recognition andlabeling of the species. Similarly, face recognition through imagecapturing has been widely used. On this basis, interpretingmicro-expressions of recognized faces is widely used. However, aslightly raised mouth corner, a slightly frowned brow, and short-termtwitches of a facial muscle generally occupy a small percentage of anentire image and are difficult to be recognized. Therefore, the improvedsegmentation and positioning algorithms of the present application areused to improve recognition and positioning of micro-expressions, whichcan improve interpretation accuracy of micro-expressions.

Referring to FIG. 1 , FIG. 1 is a flowchart of an image processingmethod according to some embodiments of the present application. Thepresent application provides an image processing method. As shown inFIG. 1 , the method includes: In step 105, an enhanced image of a targetobject of an input image is determined based on a segmentationalgorithm, where the enhanced image of the target object includes animage in which each pixel classified as the target object is displayedin an enhanced manner. The method includes: In step 110, a positioningimage of the target object is determined by applying an integral imagealgorithm to the enhanced image of the target object.

In some examples, the enhanced image of the target object includes animage in which each pixel belonging to the target object is displayed inan enhanced manner and each pixel not belonging to the target object isnot displayed in an enhanced manner. In some examples, the enhancedimage of the target object may include an image in which a pixelbelonging to the target object is displayed with enhanced brightness. Insome examples, the enhanced image of the target object may be convertedinto a form of a mask map. In some examples, the determining apositioning image of the target object by applying an integral imagealgorithm to the enhanced image of the target object includes:calculating an integral image for the enhanced image of the targetobject converted into the form of a mask map. An integral image is amethod for quickly calculating a sum of rectangular areas in an image. Avalue of each pixel in the integral image represents a sum of all pixelsin an upper left corner of the pixel in the image. Therefore, once anintegral image of an image is calculated, a sum of rectangular areas ofany sizes in the image can be quickly calculated. In some examples, thepositioning image of the target object may be in the form of a mask mapand may be determined based on the integral image. For example, a valueof each pixel in the positioning image of the target object may dependon whether the value of the pixel in the integral image is 0. If thevalue is 0, the value of the pixel in the positioning image of thetarget object is 0. If the value is not 0, the value of the pixel in thepositioning image of the target object is 1. 1 indicates that the pixelbelongs to the target object, and 0 indicates that the pixel belongs tothe image background or a non-target object.

In the technical solution of this embodiment of the present application,whether each pixel in the image belongs to the target object or belongsto the non-target object is classified based on the segmentationalgorithm, and on this basis, the target object is positioned in theinput image. The segmentation and positioning of the target object arecombined and the segmentation algorithm and the integral image algorithmare combined, so that the positioning accuracy of the target object canbe improved.

According to some embodiments of the present application, optionally,further referring to FIG. 2 and FIG. 3 , FIG. 2 is a flowchart of amethod for determining an enhanced image of a target object of an inputimage based on a segmentation algorithm according to some embodiments ofthe present application, and FIG. 3 is an effect diagram of steps ofsegmenting a target object according to some embodiments of the presentapplication. Step 102 in FIG. 1 may further include: step 205:performing feature extraction on the input image to determine a pixelfeature graph; step 210: performing feature extraction on the inputimage to determine a context feature graph; step 215: determiningcontext relation information of each pixel based on the pixel featuregraph and the context feature graph; and step 220: determining theenhanced image of the target object according to the context relationinformation and the input image, where the enhanced image of the targetobject is generated by changing, based on whether each pixel belongs tothe target object or a non-target object, a weight applied to the pixel.

In some examples, step 205 may include: inputting the input image into adeep convolutional neural network to perform pixel-level featureextraction on the input image. In some examples, step 205 may include:inputting the input image into HRNet 18 to generate a feature graph ofeach pixel in the input image. In some examples, a feature value of eachpixel in the pixel feature graph may indicate initial classification ofwhether the pixel belongs to the target object or a non-target object.In some examples, feature values of pixels range from 0 to 255, and itmay be considered that each pixel whose feature value is higher than 128belongs to the target object and each pixel whose feature value is lowerthan 128 belongs to a non-target object. In some examples, the pixelfeature graph may be a matrix representing pixel-level features (pixelrepresentation) after the input image is calculated by a deepconvolutional neural network, and an image representation thereof maybe, for example, shown in a in FIG. 3 . In some examples, step 210 mayinclude: inputting the input image into a deep convolutional neuralnetwork to perform image block-level feature extraction on the inputimage. In some examples, step 210 may include: inputting the input imageinto HRNet 18 to generate a feature graph in which the input imageincludes a pixel block including a center pixel. In some examples, thepixel block may be determined by selecting an appropriate convolutionkernel n×n, where n is an odd number. As shown in b in FIG. 3 , a blockin the figure represents a center pixel, and pixels around the blockplus the center pixel represent a pixel block. In some examples, thefeature graph of the pixel block may be a matrix representing pixelblock-level features (object region representation) after the inputimage is calculated by a deep convolutional neural network with aselected convolution kernel. In some examples, the feature graph of thepixel block represents feature values extracted in a unit of the pixelblock including the center pixel. Similarly, the feature value of thepixel block may indicate whether the pixel block belongs to the targetobject or a non-target object. In some examples, feature values of pixelblocks range from 0 to 255, and it may be considered that each pixelblock whose feature value is higher than 128 belongs to the targetobject and each pixel block whose feature value is lower than 128belongs to a non-target object. In some examples, a feature value of apixel block may represent whether a pixel around the center pixel in thepixel block belongs to the target object or a non-target object orpossibility thereof. Herein, the pixel block feature graph and thecontext feature graph can be used interchangeably to representinformation about pixels and/or context around the center pixel in thepixel block. In some examples, step 215 may include: determining contextrelation information of each pixel based on the pixel feature graphdetermined in step 205 and the context feature graph determined in step210, where the context relation information indicates the strength ofrelation between each pixel and context of the pixel. In some examples,context relation information may be obtained by performing matrixmultiplication on the pixel feature graph determined in step 205 and thecontext feature graph determined in step 210 and applying a softmaxfunction to obtain context relation information of each pixel (pixelregion relation). In some examples, when the pixel feature graph of thecenter pixel indicates that the pixel belongs to the target object (anon-target object), and the context feature graph indicates that thecontext of the pixel also belongs to the target object (a non-targetobject), the obtained context relation information of the pixel isstrong. When the pixel feature graph and the context feature graphindicate an opposite result (for example, the pixel feature graphindicates that the center pixel belongs to the target object and thecontext feature graph indicates that a context pixel of the center pixelbelongs to a non-target object), the obtained context relationinformation of the pixel is weak. In some examples, step 220 mayinclude: determining, based on the context relation information in step215, a final classification of whether each pixel belongs to the targetobject or a non-target object, and generating the enhanced image of thetarget object by enhancing, based on the final classification, eachpixel belonging to the target object. In some examples, matrixmultiplication is performed on the context relation information (pixelregion relation) obtained in step 215 and the context feature graph(object region representation) determined in step 210, to obtain aweighted pixel-level feature graph, and the weighted pixel-level featuregraph is concatenated with the pixel feature graph (pixelrepresentation) determined in step 205 to obtain a final pixel featuregraph. In some examples, the enhanced image of the target object isgenerated by changing, based on a feature value (which in turn reflectswhether the pixel belongs to the target object or a non-target object)of the pixel in the final pixel feature graph, a weight applied to eachpixel. An image representation thereof can be shown, for example, in cin FIG. 3 . In some examples, the enhanced image of the target objectmay be generated by increasing a weight applied to each pixel whosefeature value is higher than 128.

In the present application, not only pixel-level classificationinformation, but also context classification information around a targetpixel are considered in the segmentation algorithm, and a finalclassification result of the target pixel is determined based on therelation between the target pixel and the context thereof. Contextinformation is incorporated into a segmentation algorithm to furtherimprove the classification accuracy of the target pixel, therebysegmenting the target object more accurately. A weight applied to eachpixel finally classified as the target object is changed to generate theenhanced image of the target object, so that the target object isdisplayed in an enhanced manner, thereby providing a more accurate basisfor subsequent positioning processing and further improving thepositioning accuracy of the target object. The weight may be configuredby a user. A change of weight setting can affect the enhancement effectof the target object in the enhanced image of the target object, so thatthe desired enhancement effect of the target object can be achievedthrough user setting.

According to some embodiments of the present application, optionally,the determining a positioning image of the target object by applying anintegral image algorithm to the enhanced image of the target objectfurther includes: determining an integral image according to theenhanced image of the target object; and determining the positioningimage of the target object by using the integral image.

In some examples, an integral image is calculated for the enhanced imageof the target object, as shown in a and b in FIG. 4 . In some examples,normalization is performed on the integral image to find a set ofparameters based on invariant moments of the image, so that theparameters can cancel the impact of other transformation functions onimage transformation:

img_normal=img_integral/max(img_integral (:)).

In some examples, an integral image algorithm is used to find an upperleft point and a lower right point as follows:

x_left, y_left=img_normal>low_thr

x_right, y_right=img_normal>high_thr.

The integral graph algorithm is applied to the final classificationresult obtained based on the segmentation algorithm of the presentapplication, so that the target object can be accurately positioned.

The integral image algorithm is applied to the enhanced image of thetarget object in which the target object is displayed in an enhancedmanner, so that the positioning accuracy of the target object can befurther improved.

According to some embodiments of the present application, optionally,the determining an integral image according to the enhanced image of thetarget object further includes: applying a scale factor to the enhancedimage of the target object.

In some examples, a scale factor (img_scale) is applied to the enhancedimage of the target object converted into the form of a mask map. Insome examples, during the calculation of the integral image, a redundantlength can be extended to ensure positioning accuracy in the followingmanner:

y_extend=(int)((y_right−y_left)*extend_scale_y/2)

x_extend=(int)((x_right−x_left)*extend_scale_x/2).

In an example of applying the scale factor, the following formula isused to map back to the original image based on the scale factor(img_scale) to generate the positioning image of the target object, asshown in c in FIG. 3 :

x_top=(int)(max((x_left−x_extend), 0)/img_scale)

y_top=(int)(max((y-left−y_extend), 0)/img_scale)

x_bottom=(int)(max((x_left−x_extend), 0)/img_scale)

y_bottom=(int)(max((y-left−y_extend), 0)/img_scale).

A volume of data to be processed can be adjusted by applying a scalefactor, so that an operation process can be accelerated and/or theaccuracy of the integral image can be improved according to actualneeds.

According to some embodiments of the present application, optionally,the method further includes: calculating a loss rate between theenhanced image of the target object and the input image based on a lossfunction; and feeding back the calculated loss rate to the segmentationalgorithm.

In some examples, a cross entropy loss (cross entropy loss) function maybe used to calculate a loss rate between the enhanced image of thetarget object generated in step 220 and the input image. In someexamples, the calculated loss rate represents the similarity between theenhanced image of the target object and the original input image.

A loss rate between the enhanced image of the target object output bythe segmentation algorithm and a labeled production line image reflectsthe similarity between the enhanced image of the target object output bythe segmentation algorithm and the original input image. The loss rateis fed back to the segmentation algorithm to perform supervised learningtraining on the segmentation algorithm. The accuracy of the segmentationalgorithm can be improved through continuous training and learning whilefit regression is trained.

According to some embodiments of the present application, optionally,the method further includes: updating the segmentation algorithm basedon the loss rate or the labeled production line image or a combinationof both.

The segmentation algorithm in the present application trains thecalculated loss rate or the labeled production line image or acombination of both as training data, and can continuously improve theaccuracy of the segmentation algorithm in target object segmentation ina supervised learning manner. In addition, since the training data allcomes from the real production line, the training data can cover actualneeds and be used and promoted in practice in the production line.

According to some embodiments of the present application, optionally,further referring to FIG. 5 , FIG. 5 is an architecture diagram of anetwork model of a segmentation algorithm for implementing an imageprocessing method according to some embodiments of the presentapplication. The segmentation algorithm is implemented by a deepconvolutional neural network HRNet18.

In some examples, HRNet is a high-resolution network that can maintain ahigh-resolution representation in an entire process. Starting from ahigh-resolution subnetwork as the first stage, subnetworks from highresolutions to low resolutions are gradually added to form more stages,and multi-resolution subnetworks are connected in parallel. In theentire process, multi-scale repeated fusion is performed by repeatedlyexchanging information on parallel multi-resolution subnetworks.Keypoints are estimated based on high-resolution representations outputby the network, and a network architecture is shown in FIG. 4 . In someexamples, in consideration of whether the segmentation of the targetobject depends on very high-level semantic information and a limitedvolume of real training data, the smaller model HRNet18 in the HRNetseries is selected to implement the segmentation algorithm of thepresent application.

HRNet18 maintains high-resolution features in the entire segmentationalgorithm process and facilitates accurate segmentation of the targetobject. In addition, different branches of the HRNet18 network producefeatures of different resolutions, and these features interact to obtaininformation, so that high-resolution features including multi-channelinformation can be obtained. In addition, in a case of a limited volumeof training data, selection of the HRNet18 model avoids the risk ofoverfitting and at the same time can accelerate the operation speed ofthe entire segmentation algorithm because of the small structure of themodel.

According to some embodiments of the present application, referring toFIG. 1 to FIG. 5 , the present application provides an image processingmethod, including: performing feature extraction on the input image todetermine a pixel feature graph; performing feature extraction on theinput image to determine a context feature graph; determining contextrelation information of each pixel based on the pixel feature graph andthe context feature graph; and determining the enhanced image of the tabaccording to the context relation information and the input image, wherethe enhanced image of the tab is generated by changing, based on whethereach pixel belongs to the tab, a weight applied to the pixel;determining an integral image according to the enhanced image of thetab, where a scale factor is applied to the enhanced image of the tab;and using the integral image to determine the positioning image of thetab, where the segmentation algorithm is implemented by HRNet18.

Referring to FIG. 6 , FIG. 6 is a functional block diagram of an imageprocessing system according to some embodiments of the presentapplication. The present application provides an image processingsystem. As shown in FIG. 6 , the system includes: a segmentation module605, configured to determine an enhanced image of a target object of aninput image based on a segmentation algorithm, where the enhanced imageof the target object includes an image in which each pixel classified asthe target object is displayed in an enhanced manner; and a positioningimage generation module 610, configured to determine a positioning imageof the target object by applying an integral image algorithm to theenhanced image of the target object.

In the technical solution of this embodiment of the present application,whether each pixel in the image belongs to the target object or belongsto the non-target object is classified based on the segmentationalgorithm, and on this basis, the target object is positioned in theinput image. The segmentation and positioning of the target object arecombined and the segmentation algorithm and the integral image algorithmare combined, so that the positioning accuracy of the target object canbe improved.

According to some embodiments of the present application, optionally,further referring to FIG. 7 , FIG. 7 is a functional block diagram ofthe segmentation module according to some embodiments of the presentapplication. The segmentation module 605 further includes: a featureextraction component 705, configured to perform feature extraction onthe input image to determine a pixel feature graph and perform featureextraction on the input image to determine a context feature graph; acontext component 710, configured to determine context relationinformation of each pixel based on the pixel feature graph and thecontext feature graph; and an enhanced image generation component 715,configured to determine the enhanced image of the target objectaccording to the context relation information and the input image, wherethe enhanced image of the target object is generated by changing, basedon whether each pixel belongs to the target object or a non-targetobject, a weight applied to the pixel.

In the present application, not only pixel-level classificationinformation, but also context classification information around a targetpixel are considered in the segmentation algorithm, and a finalclassification result of the target pixel is determined based on therelation between the target pixel and the context thereof. Contextinformation is incorporated into a segmentation algorithm to furtherimprove the classification accuracy of the target pixel, therebysegmenting the target object more accurately. A weight applied to eachpixel finally classified as the target object is changed to generate theenhanced image of the target object, so that the target object isdisplayed in an enhanced manner, thereby providing a more accurate basisfor subsequent positioning processing and further improving thepositioning accuracy of the target object. The weight may be configuredby a user. A change of weight setting can affect the enhancement effectof the target object in the enhanced image of the target object, so thatthe desired enhancement effect of the target object can be achievedthrough user setting.

According to some embodiments of the present application, optionally,still referring to FIG. 6 , the positioning image generation module 610is further configured to: determine an integral image according to theenhanced image of the target object; and determine the positioning imageof the target object by using the integral image.

The integral image algorithm is applied to the enhanced image of thetarget object in which the target object is displayed in an enhancedmanner, so that the positioning accuracy of the target object can befurther improved.

According to some embodiments of the present application, optionally,still referring to FIG. 6 , the positioning image generation module 610is further configured to apply a scale factor to the enhanced image ofthe target object.

A volume of data to be processed can be adjusted by applying a scalefactor, so that an operation process can be accelerated and/or theaccuracy of the integral image can be improved according to actualneeds.

According to some embodiments of the present application, optionally,still referring to FIG. 6 , the system further includes a loss ratemodule 615, configured to: calculate a loss rate between the enhancedimage of the target object and the input image based on a loss function;and feed back the calculated loss rate to the segmentation algorithm toupdate the segmentation module.

A loss rate between the enhanced image of the target object output bythe segmentation algorithm and a labeled production line image reflectsthe similarity between the enhanced image of the target object output bythe segmentation algorithm and the original input image. The loss rateis fed back to the segmentation algorithm to perform supervised learningtraining on the segmentation algorithm. The accuracy of the segmentationalgorithm can be improved through continuous training and learning whilefit regression is trained.

According to some embodiments of the present application, optionally,still referring to FIG. 6 , the segmentation module 605 is furtherconfigured to update the segmentation module based on the loss rate orthe labeled production line image or a combination of both.

The segmentation algorithm in the present application trains thecalculated loss rate or the labeled production line image or acombination of both as training data, and can continuously improve theaccuracy of the segmentation algorithm in target object segmentation ina supervised learning manner. In addition, since the training data allcomes from the real production line, the training data can cover actualneeds and be used and promoted in practice in the production line.

According to some embodiments of the present application, referring toFIG. 6 and FIG. 7 , the present application provides an image processingsystem, including:

-   -   a segmentation module 605, including:    -   a feature extraction component 705, configured to perform        feature extraction on the input image to determine a pixel        feature graph and perform feature extraction on the input image        to determine a context feature graph;    -   a context component 710, configured to determine context        relation information of each pixel based on the pixel feature        graph and the context feature graph; and    -   an enhanced image generation component 715, configured to        determine the enhanced image of the tab according to the context        relation information and the input image, where the enhanced        image of the tab is generated by changing, based on whether each        pixel belongs to the tab, a weight applied to the pixel; and    -   a positioning image generation module 610, configured to:        determine an integral image according to the enhanced image of        the tab; and use the integral image to determine a positioning        image of the tab, where a scale factor is applied to the        enhanced image of the tab.

Referring to FIG. 8 , FIG. 8 is a structural block diagram of a computersystem suitable for implementing an image processing system according tosome embodiments of the present application. As shown in FIG. 8 , thesystem includes: a memory 028 storing computer-executable instructions;and a processor 016 coupled to the memory 028, where thecomputer-executable instructions, when executed by the processor, causethe system to perform the following operations: determining an enhancedimage of a target object of an input image based on a segmentationalgorithm, where the enhanced image of the target object includes animage in which each pixel classified as the target object is displayedin an enhanced manner; and determining a positioning image of the targetobject by applying an integral image algorithm to the enhanced image ofthe target object.

In some examples, FIG. 8 is a structural block diagram of a computersystem 012 suitable for implementing an image processing systemaccording to some embodiments of the present application. The computersystem 012 shown in FIG. 8 is only an example, and should not limit thefunction and application scope of this embodiment of the presentapplication.

As shown in FIG. 8 , the computer system 012 is represented in a form ofa general-purpose computing device. Components of the computer system012 may include, but are not limited to: one or more processors orprocessing units 016, a system memory 028, and a bus 018 connectingdifferent system components (including the system memory 028 and theprocessing unit 016).

The bus 018 represents one or more of several types of bus structures,including a memory bus or a memory controller, a peripheral bus, anaccelerated graphics port, a processor, or a local bus using any one ofmultiple bus structures. Examples of these structures include, but arenot limited to, an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA bus, a video electronicsstandards association (VESA) local bus, and a peripheral componentinterconnect (PCI) bus.

The computer system 012 typically includes multiple computer systemreadable mediums. These mediums can be any available medium that can beaccessed by the computer system 012 and include both volatile andnonvolatile mediums, and removable and non-removable mediums.

The system memory 028 may include a computer system readable medium in aform of a volatile memory, such as a random access memory (RAM) 030and/or a cache memory 032. The computer system 012 may further includeother removable/non-removable and volatile/nonvolatile computer systemstorage mediums. By way of example only, the storage system 034 may beconfigured to read from and write into non-removable and non-volatilemagnetic mediums (not shown in FIG. 6 , commonly referred to as a “harddisk drive”). Although not shown in FIG. 6 , hard disk drives forreading from and writing into removable non-volatile disks (for example,“floppy disks”) and optical disc drives for reading from and writinginto removable non-volatile optical discs (for example, CD-ROMs,DVD-ROMs, or other optical mediums) may be provided. In these cases,each drive may be connected to the bus 018 through one or more datamedium interfaces. The memory 028 may include at least one programproduct having a set (for example, at least one) of program modulesconfigured to perform the functions of the embodiments of the presentapplication.

A program/utility 040 having a set (at least one) of program modules 042may be stored, for example, in the memory 028. Such program module 042includes but is not limited to an operating system, one or moreapplication programs, and other program modules and program data. Eachor some combination of these examples may include the implementation ofa network environment. The program module 042 generally performs thefunctions and/or methods of the described embodiments of the presentapplication.

The computer system 012 can also communicate with one or more externaldevices 014 (for example, a keyboard, a pointing device, and a display024). In the present application, the computer system 012 communicateswith external radar devices, and can further communicate with one ormore devices that enable a user to interact with the computer system012, and/or any device (for example, a network card and a modem) thatenables the computer system 012 to communicate with one or more othercomputing devices. Such communication may occur through an input/output(I/O) interface 022. Moreover, the computer system 012 can furthercommunicate with one or more networks (for example, a local area network(LAN), a wide area network (WAN), and/or a public network such as theInternet) through a network adapter 020. As shown in the figure, thenetwork adapter 020 communicates with other modules of the computersystem 012 through the bus 018. It should be understood that althoughnot shown in FIG. 7 , other hardware and/or software modules may be usedin conjunction with the computer system 012, including but not limitedto: microcode, device drives, redundant processing units, external diskdrive arrays, RAID systems, tape drives, data backup storage systems,and the like.

The processing unit 016 executes various functional applications anddata processing by running the programs stored in the system memory 028,such as implementing the method flow provided by the embodiments of thepresent application.

The computer program can be provided in a computer storage medium, thatis, the computer storage medium is encoded with the computer program,and when the program is executed by one or more computers, the one ormore computers can be enabled to execute the method flow and/or theapparatus operation provided by the embodiments of the presentapplication. For example, the method flow provided by the embodiments ofthe present application is executed by the one or more processors.

As time progresses and technologies advance, the meaning of mediums hasbecome more and more extensive, and the transmission path of computerprograms is no longer limited to tangible mediums, and can also bedownloaded directly from the Internet. Any combination of one or morecomputer readable mediums may be used.

The computer readable medium may be a computer readable signal medium ora computer readable storage medium. A computer readable storage mediummay be, for example, but is not limited to, electrical, magnetic,optical, electromagnetic, infrared, or semiconductor systems,apparatuses, or devices, or any combination thereof. More specificexamples (a non-exhaustive list) of the computer readable storage mediuminclude: an electrical connection with one or more conductors, aportable computer disk, a hard disk, a random access memory (RAM), aread only memory (ROM), an erasable programmable read only memory (EPROMor a flash memory), an optical fiber, a portable compact disk read onlymemory (CD-ROM), an optical storage device, a magnetic storage device,or any suitable combination thereof. In this document, a computerreadable storage medium may be any tangible medium that contains orstores a program that can be used by or in conjunction with aninstruction execution system, apparatus, or device.

A computer readable signal medium may include a data signal transmittedin a baseband or as a part of a carrier wave, and carries computerreadable program code. Such propagated data signal may be in many forms,including but not limited to an electromagnetic signal, an opticalsignal, or any suitable combination thereof. A computer readable signalmedium may also be any computer readable medium other than a computerreadable storage medium, and the computer readable medium can send,propagate, or transmit a program used by or in conjunction with aninstruction execution system, apparatus, or device.

Program code contained on a computer readable medium may be transmittedthrough any appropriate medium, including but not limited to a wirelessmanner, a wireline, an optical cable, RF, or any suitable combinationthereof.

Computer program code for performing the operations of the presentapplication may be written in one or more programming languages or acombination thereof. The programming languages include object-orientedprogramming languages such as Java, Smalltalk, and C++, and conventionalprocedural programming languages such as the “C” language, or a similarprogramming language. The program code may run entirely on a usercomputer, run partly on a user computer, run as a standalone softwarepackage, run partly on a user computer and partly on a remote computer,or run entirely on a remote computer or server. In a case of a remotecomputer, the remote computer can be connected to a user computerthrough any network including a local area network (LAN) or a wide areanetwork (WAN), or can be connected to an external computer (for example,connected through the Internet provided by an Internet serviceprovider).

Finally, it should be noted that the above embodiments are merely usedfor illustrating rather than limiting the technical solution of thepresent application. Although the present application has beenillustrated in detail with reference to the foregoing embodiments, itshould be understood by those of ordinary skill in the art that thetechnical solutions recorded in the foregoing embodiments may still bemodified, or some or all of the technical features thereof may beequivalently substituted; and such modifications or substitutions do notmake the essence of the corresponding technical solution depart from thescope of the technical solutions of the embodiments of the presentapplication, and should fall within the scope of the claims and thedescription of the present application. In particular, the technicalfeatures mentioned in the embodiments can be combined in any manner,provided that there is no structural conflict. The present applicationis not limited to the specific embodiments disclosed herein but includesall the technical solutions that fall within the scope of the claims.

What is claimed is:
 1. An image processing method, comprising:determining an enhanced image of a target object of an input image basedon a segmentation algorithm, wherein the enhanced image of the targetobject comprises an image in which each pixel classified as the targetobject is displayed in an enhanced manner; and determining a positioningimage of the target object by applying an integral image algorithm tothe enhanced image of the target object.
 2. The method according toclaim 1, wherein the determining an enhanced image of a target object ofan input image based on a segmentation algorithm further comprises:performing feature extraction on the input image to determine a pixelfeature graph; performing feature extraction on the input image todetermine a context feature graph; determining context relationinformation of each pixel based on the pixel feature graph and thecontext feature graph; and determining the enhanced image of the targetobject according to the context relation information and the inputimage, wherein a pixel of the enhanced image of the target objectcomprises weight information, and the weight information is related towhether the pixel belongs to the target object.
 3. The method accordingto claim 1, wherein the determining a positioning image of the targetobject by applying an integral image algorithm to the enhanced image ofthe target object further comprises: determining an integral imageaccording to the enhanced image of the target object; and determiningthe positioning image of the target object by using the integral image.4. The method according to claim 3, wherein the determining an integralimage according to the enhanced image of the target object furthercomprises: applying a scale factor to the enhanced image of the targetobject.
 5. The method according to claim 1, wherein the method furthercomprises: calculating a loss rate between the enhanced image of thetarget object and the input image based on a loss function; and feedingback the calculated loss rate to the segmentation algorithm to updatethe segmentation algorithm.
 6. The method according to claim 1, whereinthe segmentation algorithm is implemented by a deep convolutional neuralnetwork HRNet18.
 7. An image processing system, comprising: asegmentation module, configured to determine an enhanced image of atarget object of an input image based on a segmentation algorithm,wherein the enhanced image of the target object comprises an image inwhich each pixel classified as the target object is displayed in anenhanced manner; and a positioning image generation module, configuredto determine a positioning image of the target object by applying anintegral image algorithm to the enhanced image of the target object. 8.The system according to claim 7, wherein the segmentation module furthercomprises: a feature extraction component, configured to perform featureextraction on the input image to determine a pixel feature graph andperform feature extraction on the input image to determine a contextfeature graph; a context component, configured to determine contextrelation information of each pixel based on the pixel feature graph andthe context feature graph; and an enhanced image generation component,configured to determine the enhanced image of the target objectaccording to the context relation information and the input image,wherein a pixel of the enhanced image of the target object comprisesweight information, and the weight information is related to whether thepixel belongs to the target object.
 9. The system according to claim 7,wherein the positioning image generation module is further configuredto: determine an integral image according to the enhanced image of thetarget object; and determine the positioning image of the target objectby using the integral image.
 10. The system according to claim 9,wherein the positioning image generation module is further configured toapply a scale factor to the enhanced image of the target object.
 11. Thesystem according to claim 7, wherein the system further comprises a lossrate module, configured to: calculate a loss rate between the enhancedimage of the target object and the input image based on a loss function;and feed back the calculated loss rate to the segmentation algorithm toupdate the segmentation algorithm.
 12. An image processing system,comprising: a memory storing computer-executable instructions; and aprocessor coupled to the memory, wherein the computer-executableinstructions, when executed by the processor, cause the system toperform the following operations: determining an enhanced image of atarget object of an input image based on a segmentation algorithm,wherein the enhanced image of the target object comprises an image inwhich each pixel classified as the target object is displayed in anenhanced manner; and determining a positioning image of the targetobject by applying an integral image algorithm to the enhanced image ofthe target object.